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Foreword 



This report documents the activities in preparation for the fourth follow-up of the 
sophomore cohort of the High School and Beyond (HS&B) study in 1992. It describes the 
development and testing of the data collection systems that will be used beginning in 
February 1992. Several components of these systems are innovative and should enhance data 
quality. 

High School and Beyond is major longitudinal study within the National Center for 
Education Statistics' program. It provides a wealth of information about major transitions of 
young adults from high school into postsecondary education and the labor force. 
Comparisons with other longitudinal studies, such as the National Longitudinal Study of the 
High School Class of 1972 and the National Education Longitudinal Study of 1988 furtlier 
enhance the utility of these data. With completion of the HS&B fourth follow-up in 1992, 
and the addition of postsecondary education academic transcript data in 1993, the HS&B data 
will span 1980 (when students were enrolled in the tenth grade in secondary schools) through 
1992 (when the sample members were about 28 years old). 

The HS&B data should be available in the spring of 1993. Information about 
obtaining HS&B data will be available from Roger Herriot, OERI/NCES, 555 New Jersey 
Avenue NW, Room 400J Capitol Place, Washington, DC 20208. 
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Summary of Major Findings 



This report documents the activities in preparation for the fourth follow-up of the 
sophomore cohort of the High School and Beyond (HS&B) study in 1992. It describes the 
development and testing of the data collection systems. HS&B in 1992 is the fifth wave of 
• the longitudinal study of the high school sophomore class of 1980. This longitudinal dataset 
is extremely rich. It contains past education, labor force activity, demographic, and 
attitudinal information obtained from the 1980 sophomores themselves and from school 
transcripts, school administrators (school level information) and, in some cases, sample 
members' parents. The fourth follow-up will include respondents' accounts of their life 
events (primarily in terms of education and employment) since the last round of data was 
collected in 1986. In addition, academic transcripts will be obtained for respondents who 
have attended postsecondary institutions. 

Field Test Development Products 

• Integrated Control System (ICS), a new management tool designed for monitoring of all 
project tasks 

• Electronic archive housing all project reports, deliverables, and documentation 

• Instrument Development System (IDS), containing interconnected files of data elements, 
subelementSj response categories, justifications, CATI specifications, and documentation, 
including comments from the Technical Review Panel 

• CATI instrumentation that made use of preloaded data elements, recursive text fills, 
updatable screens suitable for conversational interviewing, an on-line coding 

• On-line interviewer monitoring, with provision for statistical sampling of interviewer 
activities and provision for remote monitoring 

• Development of simplified occupational and industry coding 

Field Test Implementation 

ITie HS&B field test was designed to test the functioning of the innovative systems in 
terms of the quality and quantity of the data collected, the burden on the respondent, the 
efficiency of the interviewer task, and the technical performance of the CATI application. 
The field test comprised two distinct data collection activities. First, iterative interviews on 
sections of the questionnaire were conducted with 105 respondents. Reinteiviews were 
conducted with 25 respondents. The goal of this phase was to refine the parts of the CAT! 
instrument so that it would work efficiently and would meet the targeted length for the 
second phase, in which the main data collection conditions were to be tested* 
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The field test utilized 109 respondents selected randomly from those eligible for tlie 
fourth follow-up. Its purpose was to test fully functioning systems, to capture the timing of 
the various sections in the CATI instrument, and to test data collection proc^ures, including 
the mailing of a personalized calendar to respondents as a memory aid. The field test also 
included an experiment to evaluate locating activities by NORC locating specialists versus the 
services of an outside vendor. At the end of all locating and interviewing activities, 80 
respondents had been interviewed. 

Results and Modifications for Main Study 

• In general, the CATI instrument worked acceptably well and the overall timuig was close 
to the target for the main survey; nevertheless, it was clear that small modifications in 
sections would make the instrument more efficient and reduce burden on respondents and 
interviewers* 

• The use of preloaded data from past rounds for verification or for correction worked 
well technically. The verification/correction task for interviewers and respondents also 
worked well, though not as efficiently as possible, specifically in the section on previous 
employment. 

• Conversational interviewing and the use of updatable screens, used to verify or correct 
the proloaded data, worked so well that its use will be extended to the introductory 
section of the CATI instrument. 

• The calendar mailed to respondents as a memory aid was not found to be useful for 
those purposes. It will be eliminated for the main study. 

• On-line FICE coding worked efficiently. The use of on-line cooing will be extended to 
the modified occupation and industry coding developed for HS&B. 

• On-line monitoring of inter/iewer activities, the statistical sampling for monitoring, and 
the remote monitoring feature worked acceptably. Minor improvements will be 
implemented for the main study. 

• Lx)cating activities were conducted more efficiently by NORC locating specialists than by 
the outside vendor. Locating of respondents will be the principal difficulty in achieving 
the targeted response rate for the main study. More resources will need to be dedicated 
to the locating activity that had been originally envisioned, and more extensive and 
expensive locating methods will need to be implemented on the main study than were 
used on the field test. 
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L Introduction 



HS&B:92 is the fifth wave of the longitudinal study of the high school S( phomore 
class of 1980. This longitudinal dataset is extremely rich. It contains past educational, 
circumstantial, and attitudinal information obtained from the 1980 sophomores 
themselves and from school transcripts, school administrators (school level information) 
and, in some cases, sample members' parents. The planned fourth follow-up, or fifth 
round, will add to the unprecedented richness of data by including respondents accounts 
of their life events (primarily in terms of education and employment) since the last round 
of data was collected in 1986. In addition, school transcripts will be obtained for 
respondents who have attended postsecondary institutions. 

The National Center for Education Statistics (NCES) of the Office of Educational 
Research and Improvement (OERI), U.S. Department of Education is conducting this 
study in partial compliance with the mandate as stated in section 406(B) of the General 
Educational Provision Act (GEPA), as amended, (20 USC, 1221e-l): 

The purpose of the Center shall be to collect, and analyze, and disseminate statistics 
and other data related to education? in the United States and in other nations. The 
Center shall (1) collect, collate, and, from time to time, report full and complete 
statistics on the conditions of education in the United States: (2) conduct and publish 
reports on specialized analyses of the meaning and significance of such statistics;.,. 

PL 100-297, (Hawkins-Stafford Education Amendments of 1988) section 300 l(k) further 
states: 

The Center shall conduct a study of a statistically reliable sample of students enrolled 
in elementary and secondary school and postsecondary education training concerning 
educational progress, intellectual development, and economic prosperity. The study 
shall collect data on participation in higher education, including enrollment, 
persistence, and attainment. 

In HS&B:92, valuable information will be obtained on issues of access to and 
choice of undergraduate and graduate educational institutions, persistence in obtaining 
educational goals, progress through the curriculum, rates of degree attainment and other 
assessments of educational outcomes, and rates of return to the individual and society. 
The issues of postsecondary persistence and achievement; of excellence, equity, and 
choice in higher education; and of the economic and noneconomic rates of return of 
such education, are among the most important of the new issues explored in the 1984 
and 1986 follow-ups of HS&B. The capacity to conduct more comprehensive and in- 
depth investigations of these issues in the forthcoming fifth round of this study is one of 
the more alluring prospects of this new addition to the HS&B database. 

The base year data collection for this sample cohort was conducted in 1980, with 
follow-ups in 1982, 1984, and 1986. As the sample members move through their twenties 
it is important to evaluate the effects of their high school experiences on their lives as 
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young adults. As the 1980 sophomore cohort moves through young adulthood, the 
longer term effects of schooling-the later consequences of program and curricular 
choices, persistence or nonpersistence in school, academic achievement during high 
school, differential effects of various kinds of private and public schooling and of more 
and less effective schools-become prominent. For example, it becomes a critical task to 
determine how many dropouts hrve returned to school or have passed through 
equivalency programs, and to determine the occupational and other outcomes associated 
with each of the aforementioned patterns of behaviors and experiences. 

While each successive wave of the HS&B longitudinal study provides continuing, 
longer-teim answers to baseline questions, each successive wave also becomes, with 
maturation of the cohort, a further baseline for posing new questions that have become 
germane with the passage of time and advancement of the cohort into a new 
developmental or life stage. HS&B data have been used to monitor the critical 
transition from high school completion to labor force entry, postsecondary education, and 
family formation. However, the basic longitudinal strategy of tracking plans, decisions, 
behavior, and outcomes through repeated measurements of the same individuals over 
time is no less powerful a vehicle for exploring post-transition issues in each of these 
domains. 

Tlie fact that now-eleven years after completion of high school-many cohort 
members will have completed one or more programs of postsecondary education as well, 
serves pointedly to remind us that the mission and value of an integrated multipurpose 
panel design such as HS&B does not diminish with the end of traditional schooling. 
Education, in one or another form, is a lifelong process; and the consequences of 
educational antecedents are ineluctably manifested throughout the life-course of the 
individuals who have passed through the formal educational system. Indeed, the fourth 
follow-up of HS&B may usefully be viewed in the context of the Center's strong 
affirmation of the need for comprehensive statistical investigation of educational issues to 
fulfill the NCES charter to report on the condition and progress of American education 
in all its aspects. The inclusion of adult literacy and lifelong learning as one of six 
national education goals enunciated by the President serves to underline the importance 
of studies such as HS&B:92, which can help not only to measure achievement of certain 
aspects of such performance goals but can also help to uncover the relationship between 
the effects being measured and their principal determinants. This is a principal strength 
of such studies in comparison to even larger scale cross-sectional assessments of 
educational progress. 

In the fourth follow-up, the student interview will emphasize five areas pertinent 
to 1980 high school sophomores now in their middle rwenties: undergraduate and 
graduate access and choice, persistence, progress through the curriculum, attainment and 
outcome assessment, and rate of return. These thematic topics serve to unify and 
complement other current and future NCES-sponsored postsecondary longitudinal 
studies. HS&B:92 is particularly well suited to examine each of these themes because: 
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(1) many items in prior rounds are related to these themes, thus providing a temporal 
context, and (2) the age of the respondents places them at a time when new information 
concerning the issues addressed by each of the themes would provide invaluable insights 
into the effects of secondary and postsecondary education. 

The remainder of this report contains a description of the HS&B fourth follow-up 
field test. The field test was unique in a number of ways. One unique feature was that 
the field test procedures were tested on a small sample of main study respondents 
sample, rather than on a field test sample. This was necessary in order to adequately 
test procedures used to link information from prior rounds to the present interview. A 
second unique feature of the field test concerned the fact that interviewers were 
presented with several new data collection challenges, such as unscripted interviewing 
and on-line coding. Finally, a number of innovative features were developed for both 
project management and data collection. 

In the sections that follow, we describe the field test systems and procedures in 
detail, and we present the results of the field test data collection effort. The second and 
third sections describe and evaluate the HS&B Integrated Control System (ICS), a new 
management tool developed especially for this project. Sections four and five discuss 
instrument development, including both substantive issues and technical issues associated 
with the computer assisted telephone interviewing (CATI) system. 

Sections 6, 7, 8, and 9 discuss the data collection activities in the field test. The 
field test comprised two distinct data collection activities. First, iterative pretests on 
sections of the questionnaire were conducted. Results from the pretests were used to 
develop field test data collection procedures and to refine the field test CATI 
instrument. Second, a formal field test was conducted on 109 respondents in order to 
test the functioning of the full range of systems, and to obtain valid estimates of the 
instalment length and the extent of locating. The final sections of the report (sections 10 
and 11) evaluate the CATI system and make recommendations for the main survey. 

2. Description of ICS 

The application of computer technology has allowed the development of 
increasingly complex survey designs. Samples can be constructed in more intricate ways, 
and each component can be tracked separately to ensure adequate response rates; 
instalments are designed for computer-assisted administration which will automatically 
follow branching and verify the consistency of data collected; field periods have been 
shortened making it necessary for more people to be involved in the survey process. All 
too frequently, the people working on a project are specialists who see no need to 
become acquainted with the full range of project activities, or who lack access to 
complete information even if they want it. All of this demands still more complex 
management tools to ensure that the many pieces are moving in concert. 

3 
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NORC has developed an integrated management system, running on our local 
area network to accomplish this task. Rather than making a huge investment in software 
development, the system is composed largely of commercially available products. This 
provides a number of advantages: 

• As new versions of components are developed, they can be quickly integrated into 
the system 

• Continuous improvement of the system components is assured by competition in 
the marketplace 

• When problems occur, the manufacturer's staff is responsible for finding quick 
solutions 

• Training is readily available 

The High School and Beyond fourth follow-up Integrated Control System (ICS) is 
such an integrated management system. It provides a single entry point for HS&B users 
to obtain the current status of the study, to access past infonnation on the project, view 
the data collection instruments £uid monitor the progress of instrument development, 
review the project schedule, and conduct some data analysis. It must be emphasized that 
the development of the ICS is ongoing and will be a dynamic process. There is not a 
complete plan for the system-nor would it be appropriate to have one. The concept 
itself is still in development. The goal is the iterative, experimental development of a 
system that will meet the needs of the project, while staying within the project budget. 

Major components are: 

• The instrument development database 

• Tlie computer-assisted telephone interviewing system CATI) 

• An on-line interviev/er monitoring module 

• An electronic codebook with data from prior rounds 

• An electronic codebook with current data (starting with data collection) 

• A data analysis tool 

• Status reports (management, budget, data collection, monitoring, etc.) 

• Current schedule and resource plans 

• A document archive 

Initial entry to all ICS modules is through a menu produced in the Novell 
Menuing System on a Novell network. Users simply highlight the desired module using 
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the arrow keys and launch the application by hitting "Enter." Through the menu, users 
also have access to standard applications such as WordPerfect and to a number of help 
screens. The menu also provides access to the following Modules. 

Instrument development module. The data collection instrument has been 
developed with the help of a commercial text database called askSam. Tht database 
allows the development of a series of interconnected data files containing data elements; 
subelements; questions; range, type, and consistency checks; response categories; 
interviewer question-by-question instructions; data element, and question justifications; 
and instrument development history, notes, and documentation. Most of the linkages are 
bi-directional. As a result, it is possible to navigate through the system starting with any 
item and find all related items. This module has been used by project staff to 
completely define. the field test data collection instrument. The files generated will be 
loaded into the electronic codebook and the CATI module. 

CATL The field test CATI system is controlled by a commercial product, 
AutoQuest. Once it is loaded, the field test CATI instrument is available through the 
ICS. During instrument development, this was the instrument version currently under 
testing. During field test data collection, a copy of the final field test CATI instrument 
was accessible, allowing users to conduct mock data collection, thus providing another 
level of instrument testing. To protect the actual field test data, any data generated by 
the mock interviews were be stored in a separate database. 

On-line interviewer monitoring. An on-line monitoring module was developed to 
allow monitors to enter monitoring codes into the computer, rather than recording them 
on paper. Use of this program will enable the provision of timely data on interviewer 
performance. 

Electronic codebook. The electronic codebook has been developed by NCES staff 
and staff from Pelavin Associates. As mentioned above, information from the instrument 
development module database is loaded into the electronic codebook. On a periodic 
basis during data collection, completed cases are extracted from the AutoQuest database 
and used to update the current data codebook database. 

Data analysis. A standard commercial data analysis package {SPSS-PC and/or 
SAS'PC) is included in the ICS. Because most users will not have the capacity on their 
own PC's to do very large or sophisticated analyses using these packages, the ability to 
queue analyses jobs on a 486-based PC is provided through the ICS. Batch processing 
software, PS-Batch Processor, will be running on the 486. It accepts any MS-DOS batch 
file and sends a message to the submitter when the job is completed. The batch 
processor can be used for other large jobs from within the ICS. 

Document archive. The document archive has been set up in a commercial' 
product, SoftSolution. This product captures a profile of information on each document 
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including such items as document name, date, author, and location-whether stored 
electronically or as hard copy. The resultant database is fully indexed and can be 
searched using any combination of variables. Security is built in, potentially allowing 
different users varying levels of access (although this has not been done at present). 
When a document is selected for retrieval, the system launches the appropriate 
application (e.g., WordPerfect or Lotus l-l-S) and loads the retrieved document. 
Documents are submitted for inclusion in this system through the NORC electronic mail 
system (cc:Mail). A mailbox has been established for the project and all substantial 
communications are copied to that mailbox. The system operator regularly empties the 
mailbox and feeds documents into SoftSolution. All project staff, whether at NORC, 
NCES, Pelavin, or elsewhere, have access to the electronic mail system. Users not 
housed at NORC or who are travelling use a package called caRemote over the 
telephone to send and receive messages with other project staff and to send items to the 
document archive. 

Reports. Reports generated on a regular basis are available for viewing or 
priiiting through the ICS. A simple shareware list product is used to display these files 
on the screen. The reports are also stored in the document archive. 

Current schedule and resource plans« NORC uses Project Workbench (PW) to 
plan project activities and schedule resources. Standard reports are generated and are 
available for viewing or printing through the ICS. They are also stored in the document 
archive. ICS users are able to use PW through the ICS to explore the effects of possible 
changes to the project plan on a duplicate database. 

3« Evaluation of ICS 

Speciflcations. The design of the ICS expanded from the notion of a high- 
powered data management system to an executive information system. The ICS was 
built in modules, and each module of the prototype was tested as it was developed and 
during a site visit. 

Hardware/software decisions. Decisions about the hardware and software 
components of ICS were driven by considerations of availability and simplicity. Some 
components, such as AutoQuest, WordPerfect, and Project Workbench, are standard NORC 
tools. One challenge was determining the best method of permitting access to the 
different functions of the ICS. A unifying Novell menu was selected after hot-key and 
WINDOWS options were rejected as complex and error-prone. 

Other software was needed for the documentation repository and th ^ 
questionnaire development functions of the ICS. SoftSolution was chosen i jv the former. 
It was inexpensive, and already licensed to NORC. In retrospect, is proved to be 
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insufficiently flexible to provide a long-term solution to repository problems. It also 
requires a fair amount of (unbudgeted) maintenance to be useful. 

A key function of the ICS was the Instrument Development System (IDS). The 
major problem presented by the questionnaire development function was creating a 
relational database of free-form text. The askSam program was chosen because staff had 
some experience in its application from a previous project. In practice, askSam's 
functionality was not fully utilized. Instrument development staff tended to edit text using 
WordPerfect and then import the file into askSam^ rather than working directly in the 
database. 

A program was developed to export and convert questionnaire specifications into 
AutoQuest code. The program was able to generate about 55 percent of the code 
necessary to build the CATI instrument. While the effort was only partially successful 
due to limited time and resources even the partial success saved considerable time and 
expense in instrument development. As this component develops, a greater percentage 
of the AutoQuest programming will be automated. 

An as yet undelivered function of the ICS is the capacity to run SAS and SPSS-PC 
analyses on questionnaire data. A major hurdle is the memory requirements of the 
statistical packages. A LAN batch processing system was developed as a step in providing 
this capacity. 

4. Instrument Development 

Preliminaries. Preliminary work on the HS&B fourth follow-up field test 
instrument began before the contract was awarded. Based on past rounds of HS&B, and 
on assessments of current and future information needs, NCES constructed a list of "data 
elements", or conceptual categories, which were to form the basis for the questions asked 
of HS&B fourth follow-up respondents. The data elements were presented with examples 
of questions used in various NCES studies, including the HS&B third follow-up, the 
National Postsecondary Aid Study (NPSAS), the Beginning Postsecondary Survey (BPS) 
and the National Longitudinal Study of the High School Class of 1972 (NLS-72). 

As a first step in instrument development, HS&B:92 project staff examined the 
recommended items associated with each of the data elements and rated them for their 
appropriateness. Results of the ratings were sent to the COTR who evaluated NORC 
recommendations and added his own. A final list of data elements and associated 
questions was constructed and sent to member^ of the Technical Review Panel (TRP) 
with instructions to send preliminary feedback lo NORC. A number of members 
responded and their comments were assembled into a document for discussion at the 
TRP meeting which was held in Washington, D.C. in February 1991. 
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TRP/IRB reviews* This section documents the review of data elements conducted 
by the HS&B TRP during its meeting in February 1991. It also documents the review of 
the NCES Interdivisional Review Board (IRB). The first TRP Meeting for HS&B:92 
was held by NORC on February 6 and 7, 1991, at NCES in Washington, D.C Fifteen 
of 17 panel members attended the sessions. In addition, a number of staff members* 
from NCES participated in the discussions. On February 8, the NCES IRB reviewed the 
data elements for HS&B:92. 

One important issue examined by the TRP concerned cleaning and verification of 
education and employment data coUectecfiiuring prior rounds of data collection. For 
employment data it was agreed that it would not be cost effective to clean each 
employment spell collected in prior rounds. It would rather be more useful to analysts, 
and an easier task for respondents and interviewers, to collect aggregate employment and 
earnings data for the years of prior data collection. The same aggregate data will be 
collected for the years 1986-1992 in addition to data on some individual employment 
spells since 1986. For education data it was agreed that each prior education spell would 
be verified, and all new spells would be collected for the years 1986-1992. 

The TRP recommended dropping specific items relating to the teaching 
profession. It was felt that other surveys are more appropriate for addressing issues 
related to the teaching profession, and that in the HS&B cohort there are not enough 
teachers to warrant collecting special items. The TRP panel recommended dropping 
questions about specific sources of financial aid since panel members felt that other 
datasets, such as NPSAS, address the issue more comprehensively and accurately than 
would be possible with HS&B:92 data. Aggregate questions about education loans will 
still be included. 

Additional points regarding education contributed by the TRP included the need 
to distinguish proprietary schools, from nonprofit private schools, and the need to 
distinguish education spells where students attended half-time or more as opposed to 
either full-time or less than half-time. The IRB suggested that reasons for leaving 
postgraduate schooling in addition to collecting the index of progress be considered as 
items. 

The panel discussed the importance of addressing the issue of employment-related 
training, employer-provided or otherwise. Both the TRP and IRB were interested in 
expanding the section on family and children. The TRP suggested an assessment of the 
respondent's financial support for others, such as parents or grandparents, who would 
not qualify as dependents. The IRB suggested questions about day care for children 
under five and about the nature and grade-level of schooling for children age five or 
more years. 
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After considerable discussion, the TRP recommended dropping the locus of 
control question, but retaining the questions on values and political participation, with 
modest enhancement of the political participation question. 

The TRP was also interested in assessing indebtedness. Toward that end, data 
elements were added to ascertain mortgage balance on homes or condominiums, and the 
make, model, and year of respondents's automobiles as well as the remaining automobile 
loan balance. The TRP suggested gaining greater disaggregation regarding 
race/ethnicity, especially with regard to black respondents, and recommended including 
items asking about the nature and amount of interracial and interethnic experience. 

Description of the High School and Beyond Field Test Instrument Development 
System (IDS). Another step in the HS&B:92 instrument development process involved 
the creation of an Instrument Development System (IDS) designed to achieve the 
following goals. The system was designed to be one that allows the linking of survey 
questions to data elements and associated documentation, such changes resulting from 
the OMB forms clearance process, and comments by the TRP and the NCES IRE. 

The IDS consists of four parts: 

1. A database that stores the attributes of each item in the instrument 

2. A system for editing and querying the item database 

3. A CAll Description Language (CDL) that allows non-programmers to 
describe CATI questions in terms of a small number of question attributes 

4. A translator for converting the entries in the database into source code 
that is acceptable to NORC's CATI system 

For instrument development we created five askSam question files that can be 
linked using the database package. These files are: 1) data element justifications file, 2) 
data subelement text file, 3) TRP notes file, 4) documentation file, and 5) question text 
file. The IDS can be used with files 1, 2, and 3 to quickly and easily see relationships 
between justifications, data elements, subelements, and TRP comments. The askSam 
program includes a full feature query language that is capable of performing queries on 
these elements. 

It was our intention that the IDS should aid in the creation of a working 
HS&B:92 CATI instrument, in such a way that the path from data element, through 
TRP, NCES, and OMB recommendations, to question text and, finally, CATI screen was 
clearly documented. Files 4 and 5 are critical to this linkage. File 4 provides the 
instrument development team with a place to document ways in which data subelements 
were altered as they were turned into questions. File 4 also provides a vehicle for 
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documenting changes suggested by the TRP, NCES, or 0MB. File 5 is a file that 
contains questions and response categories, along with symbols recognized by the CATI 
system NORC uses. Output from file 5 provides a basis for the input file to the CATI 
program. 

Linkage of variables to the data elements has been extended to the electronic 
codebook and data file through variable naming. Consistent with HS&B convention, 
each variable name begins with "¥4" to indicate the fourth follow-up, young (i.e., 1980 
sophomore as opposed to 1980 senior) cohort. Following this are three digits indicating 
to which data element and subelement (i.e., subcategory within data element) the item 
belongs. Finally, the variable name is made unique by following the fifth digit with 
sequential numbering within data subelement. The naming convention completes the 
loop between data elements and variables that will ultimately appear on the data file. 

Link to CATL A translator component of the IDS stores the description of each 
question in an askSam database using a simple syntax. The translator is used to convert 
the item description into the language supported by the CAT! system used by NORC. 
The current version of the translator is capable of producing 55 to 60 percent of a 
finished CATI questionnaire from items stored in the IDS database. The remaining 40 
to 45 percent must be done by a CATI programmer. Future versions of the translator 
and language will bring the efficiency up to about 75 percent. 

Conversation mode interviewing. Data collection for this newest round of High 
School and Beyond represents an important departure from the four previous rounds- 
the shift from a self-administered mail questionnaire to CATI. CATI has a number of 
advantages over mail questionnaires that we believe will result in improved data quality. 
However, the concern was that a shift from a visual, respondent-paced method of data 
collection (the mail questionnaire) to a verbal, interviewer-paced method (CATI) would 
affect responses. 

An important step taken to the collection of high-qualit>' and comparable data 
was the careful attempt to make the CATI interview as natural a process as possible. 
We tried to accomplish this by writing questions that seemed to flow naturally, as if the 
interviewer and respondent were engaging in a conversation. Two techniques were used 
to accomplish this goal: (1) whenever possible, questions were written to be short with 
informal wording, and (2) whenever possible, questions were structured so that responses 
were captured naturally, as they would in a conversation. As an example of both of 
these techniques, rather than a question that asks "Tell me whether each of the following 
people lives in your household; your father, your mother, your spouse, etc.", we wrote 
"Who lives with you?". Rather than asking you to read each response category in this 
case, you are asked to code the respondent's free response to the question, ask 
clarification questions when necessary, and prompt with "who else?", when the 
respondent stops giving responses. We hope this conversational form of question writing 
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and response capturing will result in shorter, more pleasant, less tedious interviews with 
high-quality data. 

In addition, in this round of High School and Beyond, employment, and income 
data from past rounds will be verified and cleaned. Because respondents' ability to 
recall their pa^ t experiences may be affected by both the passage of time and the use of 
a CATI formal, respondents are provided with a calendar that tracks employment and 
education history as it appears in the data-file from previous rounds. These were sent to 
field test respondents, accompanied by a letter forewarning respondents of the interview, 
asking them to check the information on the calendar for accuracy and completeness 
before the interview, and asking them to have the calendar on hand so that it could be 
referred to during the interview. 

Interviewers had the same information in the form of CATI summary screens 
described earlier. It was difficult and awkward to write scripted questions for verifying 
the information on the summary screens. Therefore, we asked interviewers to go over 
the information item-by-item with the respondent, without having a definite script, 
requesting only that their style is a natural, friendly, and pleasant one and that it includes 
asking about missing information. 

As befits sound survey practice, the onus of conversational mode interviewing still 
rests upon the designers of the instrument. For the most part, interviewers were 
instructed to read questions and responses exactly as written. However, for the field test, 
we asked interviewers to keep track of question wordings that they found to be awkward, 
and to report these and suggested rewording to their supervisors, who would in turn 
advise project staff responsible for instrument development. Some of these 
recommendations are presented in other parts of this report. 

Development of industry coding scheme. The coding scheme for industry 
developed for the field test and proposed for use on the main survey was a simplified 
version of the U.S. Department of Commerce, Bureau of the Census Classified Index of 
Industries and Occupations 1970, and the U.S. Department of Commerce, Bureau of the 
Census Alphabetical Index of Industries and Occupations 1970, which had been used on 
previous rounds of High School and Beyond. To simplify the coding task, the major 
headings were used as the coding categories with one exception. Manufacturing was split 
into Durable and Nondurable Goods. These headings are reproduced below. 

Agriculture, Forestry^ and Fisheries 

Mining 

Construction 

Manufacturing— Durable Goods 
Manufacturing— Nondurable Goods 

Transportation, Communications, and Other Public Utilities 
Wholesale Trade 
Retail Trade 
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Finance, Insurance, and Real Estate 
Business and Repair Services 
Personal Services 

Entertainment and Recreation Services 
Professional and Related Services 
Public Administration 
Industry Not Reported 

For researchers who wish to code at a more detailed level, the entire verbatim 
response will also be reported in the data file. 

Development of occupation coding scheme. The coding scheme for occupation 
coding used in the field test, and proposed for use on the main sur/ey, was adapted from 
the following question in the High School and Beyond third follow-up questionnaire. 

Variable: TY15A DESCRIBE JOB AT 30 YRS OLD 
Module: 3F4 Position: 31-32 

Write in here the name of the job or occupation that you expect or plan to have when 
you are 30 years old. Even if you are not at all sure, write in your ONE best guess. 
Which of the categories below comes closest to describing that job? 

CLERICAL such as bank teller, bookkeeper, secretary, typist, mail carrier, ticket agent 

CRAFTSMAN such as baker, automobile mechanic, machinist, painter, plumber, telephone installer, carpenter 

FARMER, FARM MANAGER 

HOMEMAKER (without other job) 

LABORER such as construction worker, car washer, sanitary worker, farm laborer 

MANAGER. ADMINISTRATOR such as sales manager, office manager, school administrator, buyer, restaurant manager, 
government official 

MILITARY such as career officer, enlisted man or woman in the Armed Forces 
OPERATIVE such as meat cutter, assembler, machine operator, welder, or truck driver 

PROFESSIONAL such as accountant, artist, registered nuise, engineer, librarian, writer, social worker, actor, actress, 
athlete, politician, but not including school teacher 

PROFESSIONAL such as clergyman, dentist, physician, lawyer, scientist, college teacher 

PROPRIETOR OR OWNER such as owner of a small business, contractor, restaurant owner 

PROTECTIVE SERVICE such as detective, police officer or guard, sheriff, fire fighter 

SALES such as salesperson, advertising or insurance agent, real estate broker 

SCHOOL TEACHER such as elementary or secondary 

SERVICE such as barber, beautician, practical nurse, private household worker janitor, waiter, waitress 
TECHNICAL such as draftsman, medical or dental technician 
NOT WORKING 




Third follow-up item frequencies for this question were reviewed to determine 
which categories needed to be further subdivided. It was decided to add codes (from the 
list of standard occupation codes) within the following larger categories: 



TABLE 4.1 Frequencies for large occupation code categories 


Code 


Frequency 


Percent 


Label 


1 


1213 


9.0 


CLERICAL 


6 


1638 


12.2 


IVIANAGER, ADMINISTRATOR 


9 


2823 


21.0 


PROFESSIONAL TYPE A 


10 


948 


7.1 


PROFESSIONAL TYPE B 


11 


1023 


7.6 


PROPRIETOR OR OWNER 


16 


902 


6.7 


TECHNICAL 



The augmented list was then reviewed by several labor economists at the 
University of Chicago* The final list is reproduced below along with the mapping 
between these codes and the original SOC codes. 
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TABLE 4^ - Proposed Occupation Codes 



gp<|f DfscriDtioo 


Bxoiuule^ 


01 




secretary, typist, Hie clerk, receptionist 


02 




bookkeeper, bank teller 


03 




ticket agent, mail carrier, etc. 


04 


Cnttsmm 


baker, automobile mechanic, machinist, painter, plunaber, carpenter 


05 


FanMfr/Fann MAuager 




06 


Hoai<auk«r (wkMout oilier Job) 




07 


Labortr: 


construction worker, car washer, sanitary worker, farm laborer 


OB 


Maaagtr/AdimuLvtrator-- 








sales manager, buyer 


09 


MaNager/ Adiumlstnitt ir- 






Govenuueni: 


local, state or fedcr?vl 


10 


MaMger/Admiaisirfitor- 






Retail/Htttpaality 


store manager/assistant manager, restaurant manager/assistant man 






hotel manager/assistant manager 


11 


Manager/ Admiuisirator- 






MaiMifacturuif 






/Coastmction 


line supervisor, quality control supervisor 


12 


Mauager/Adiuluistrator-' 






OUier 




13 


Militaiy: 


career officer, enlisted personnel 


14 


Skilled Operative: 


meat cutter, assembler, machine operator, welder, truck driver 


15 


ProfM6aoaal^Art«/ 






Enlertaioiuem/Media 


actor/actress, artist, writer, athlete 


16 


Profeuioiial^-Medicui: 


registered nurses, medical assistant, social worker 




DOES NOT INCLUDE PHYSICIANS 




17 


ProfesstOf2Al— Eugiueer 


mechanical engineer, electrical engineer 


18 


Profe&sioual^-Physicuui: 


dentist, neurologist, psychiatrist 


19 


Profefifiioual-Legot 


lawyer, judge 


20 


Prof MsioMal-OUier 


clergyman, scientist, college professor 


21 


Propneior/Owiier-* 






Retail/ Hospttality: 


store ov,*ncr, restaurant owner, hotel owner, motel owner 


22 


Propneto r/Owuer- 






Mauufacturiug/ 






Co&Mructioa: 


construction contractor 


23 


Pro prieto r/Owner-Other 




24 


Protective Services: 


detective, police officer, fire fighter 


25 


Sales: 


salesperson, advertising or insurance agent, real estate brokci 


26 


School Teacher 


elementary or secondary school teacher 


27 


Service: 


barber/beautician, janitor, waiter, waitress, practical nurse 


28 


Techuical- 






Computer related* 


computer programmer, co .iputer technician 


29 


Technical- 






Nou-couiputer related 


draftsman, medical/dental technician 


30 


Not Workiiig 
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Description of High School and Beyond Field Test CATI systems 



The CATI program used by NORC for the High School and Beyond fourth 
follow-up field test is A' *oQuest, published by MicroTab of Australia. AutoQuest 
provides the following features: 

• Display of interviewer instructions, survey questions, and response categories 

• Display multiple questions per screen 

• Screen displays of "fills" in text based on answers to prior questions or based on 
data from previous rounds 

® Validity checking based on range, type, and comparison to previous answers 

• Entry of open-ended or verbatim text 

• Branching or skipping based on previous answers and/or on preloaded data 

• Capacity to suspend an interview and restart it at another time 

• Capacity to review and change a previous response 

• A system for call scheduling (TNMS) 

The questionnaire for HS&B fourth follow-up made innovative use of several of 
these features. For example, in order to present a more conversational style of 
interview, related groups of questions were presented together on one screen wherever 
possible. The effect was a more streamlined application. Also, response categories were 
frequently presented as point-and-shoot style menus rather than as lists of text with 
codes. Finally, over 100 data items were preloaded from previous rounds. These were 
often presented to the interviewer on a single screen. This form-based presentation 
allowed the interviewer to efficiently verify each piece of information and, if necessary, 
to replace erroneous information or fill missing fields. 

The interview was implemented as two AutoQuest questionnaires. .The first 
questionnaire was used to locate and verify the identity of the respondent. Final 
outcome codes were collected in this questionnaire. The second questionnaire contained 
all of the survey questions. The two questionnaires were linked so that with a few key 
strokes an interviewer could move easily between them. The primary advantage of this 
arrangement was one of performance. The most frequently used questionnaire was the 
locating questionnaire, which could quickly display case information. The larger, slightly 
slower instrument was not accessed until the interviewer had actually contacted the 
respondent and had obtained the respondent's consent to proceed with the interview. 

Call scheduler. The call scheduling system that NORC normally uses for CAT! 
studies, AutoQuest ll^yiS, was not used in the HS&B field test for two reasons. First, 
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with small samples call schedulers do not perform efficiently. Second, a small sample 
does not provide a good test of the system as it functions under a heavy load. For these 
reasons, it was decided to test the TNMS separately. 

A new release of AutoQuest /TNMS will be used for the main survey of HS&B. 
Several of its new features were designed specifically with HS&B in mind. The tests that 
will be made to insure that the TNMS will meet HS&B main survey needs are described 
below. 



Testing will be done in three phases. The first phase will test the basic features 
of TNMS, both new and old. The second phase tests the scheduling algorithms, and 
evaluates TNMS performance under heavy load. The last phase examines side effects, 
the effects of unexpected keystrokes, and provides another load test. 

Phase 1. For phase 1 testing, the existing HS&B questionnaire will be attached to 
the new version of AutoQuest /TNMS. All requested changes will then be made to these 
questionnaires. Finally, changes that allow access to new features will be made. This 
will test the downward compatibility of the new software. 

Next, the basic features of the new version of AutoQuest /TNMS as they relate to 
our questionnaires will be tested one by one. These tests will include: 

• An examination of completed cases for accuracy in data capture 

• An examination of suspended (breakoff/retrieval) cases for accuracy in 
data capture 

• A recreation of previous problems to see if they still exist 

• A test of retrieving cases based on alternate indexes (new feature) 

• A test of jumping from one questionnaire to another (new feature) 

• An examination of the accuracy of TNMS reports 



A database of these and many other tests has been prepared and will be used to 
record all test outcomes. Besides capturing test results, information in the database will 
be used to verify that all tests were actually made. 

Phase 2. In Phase 2, two thousand dummy cases will be loaded into a very simple 
questionnaire. Each case will be loaded with only one piece of information, a value 
indicating how long to pause before storing the case. The pause times will be calculated 
to reflect our estimated usage for the main survey. Five testers will use 20 machines to 
test the behavior of the system under load. The testers will wait for the indicated 
period of time and then return the case to the system with a new disposition. The 
following observations will be made: 
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• Are appointments delivered to a station on time? 

• Does the scheduling strategy hide cases in any way? 

• Does every case have a fair chance of being worked? 

• Are problem cases (such as refusals, language problems, etc.) diverted to 
the appropriate specialist? 

Phase 2 will also be used to decide what the minimum number of free cases should be 
for the system to function smoothly. 

Phase 3. Phase 3 test^ the TNMS under field conditions. In Phase 3, five 
thousand dummy cases will be loaded into the questionnaire described in Phase 1, 
Twenty interviewers will enter cases all day over a number of days. A programmer will 
be on hand to record problems that arise. 

6. Pretest 

Field test data collection occurred in two parts: a pretest, in which the instrument 
was tested in modules, and a full-scale field test, in which the entire instrument was 
tested, along with the online coding, online monitoring, and locating procedures. The 
pretest and field test samples were drawn from among main study respondents. The 
field test CATI instrument was divided into three modules. Module 1 consisted of all 
questions on postsecondary education, Module 2 consisted of all questions on 
employment after high school, and Module 3 consisted of all other questions (e.g. 
demographics, values, political participation, etc.). 

The pretest was divided into five phases. Iterative testing within four of the 
phases was conducted. For these four phases, part one constituted an initial test of the 
module, while part two was the test of modifications made as the result of interviewer or 
respondent conmients about the module. Iterative testing was conducted for each 
module and then for the three modules together. 

Sample design and selection. Because the focus of the pretest was on testing 
specific sections of the questionnaire-in particular the sections concerned with education 
(Module 1) and employment (Module 2)-we thought it was important for the pretest 
sample to include cases with differing employment and educational experiences. To 
meet this objective, we stratified respondents in the third follow-up sample into four 
groups: Group 1 included cases who reported two or more jobs and at least one 
education spell in the third follow-up; Group 2 included cases who reported two or more 
jobs but no education spells in the third follow-up; Group 3 included cases who reported 
one or more education spells but no jobs; Group 4 included those who reported no 
education spells and fewer than two jobs. Table 6.1 below shows how cases from these 
four groups were allocated across the five phases of the pretest. The sample sizes are 
given in terms of the number of selected cases, not the number completed. 
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TABLE 6.1 - Distribution of Cases by Group and Pretest Phase 


Phase 


Modules 


Sample Size 


Tested 


Group 1 


Group 2 


Group 3 


Group 4 


Phase 1 


Module 1 

(Postsecondary Education) 


10 


10 


10 


10 


Phase 2 


Module 2 
(Employment) 


10 


10 


10 


10 


Phase 3 


Module 1 & 2 


7 


8 


8 


7 


Phase 4 


Module 3 

(Other Information) 


8 


7 


7 


8 


Phase 5 


Modules 1-3 


15 


15 


15 


15 



Cases were selected randomly within each group and then assigned randomly to 
one of the five phases. Pretest plans called for completing interviews with about half of 
the selected cases, or a total of 100, with 25 in each of the four groups. To reduce the 
cost of the pretest, the pretest cases were selected from among the third follow-up 
respondents whose addresses had been updated or verified in 1989, during the most 
recent effort to obtain locating information. There were about 5,900 such cases. 

Table 6.2 shows the number of respondents tested at each phase of the pretest. 
Phase 3 was combined with Phase 5, and the targeted number of cases were completed 
in all phases except Phase 2. Part 2 of Phase 2 was canceled when the first part revealed 
no problems with the employment module. 



TABLE 6^ - Number of cases from phases of the pretest 




Part 1 


Part 2 




n 


n 


Phase 1 


10 


10 


Phase 2 


10 


0 


Phase 3 


8 


7 


Phase 4 


8 


7 


Phase 5 


45 


0 
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Reinterviews. Reinterviews were conducted with 25 of the Phase 5 respondents. 
The reinterviews were designed to elicit respondent comments concerning various aspects 
of the interview-such as length of interview and usefulness of the calendar-and to assess 
reliability of responses to selected items. Results are listed below: 

• Nineteen (76%) of the respondents received the calendar and advance 
letter sent to them before the interview date. 

• Of the nineteen that received the calendar before the interview date, only 

9 (47%) found the calendar to be helpful during the intemew. The other 

10 respondents indicated that though they had received the calendar they 
did not have with them durmg the interview. 

• Ten respondents (40%) said the interview was too long. However, only 2 
of these said that the length affected their responses. 

• Only 3 respondents found any portion of the interview to be offensive. In 
each case, respondents objected to a single item concerning either annual 
income or race. 

Interviewer debrieflng. At the conclusion of interviewing for Phase 5 of the 
pretest a formal interviewer debriefing was held to elicit interviewer comments on the 
data collection instrument. Interviewers and project staff discussed each question in the 
instrument to determine which ones were functioning properly and which ones required 
revision. Listed below are the items that were revised based on feedback from the 
interviewer debriefing. 
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TABLE 63 - Results of interviewer debriefing 


Question 
Number 


Subject Matter 


Change 




Introduction to questions concerning 
participation in extracurricular activities at 
postsecondary institutions. 


Question was re-worded to include 
reference to most recent/current 
postsecondary institution. 


Y4319A 


Employer-provided training received by R 
during the past twelve months 


Question was re-worded for clarification. 
New wording is "During the last twelve 
munLUo uavc yuu rcLCivcu any ciiij^iuyci- 
provided training?" 


YdAOl A 

I 'rOUi/V 


OUluLCd Ut luLUUlC 


r*^iif»cti Pkn u/ac chrkrff»nf»H tn "T nr\\Jl/ 

V/UColiUil Woo ollUI LCllCU l\J L Will IIUW 

read a list of different sources of income. 
Please tell me if you received any income 
from each of these sources during the 
past twelve months." 




iVldrUal oldlUd 


/\ 1 Copunbc cuLCj^ury wab ciuucu lui 
people who are separated. 


Y4324D 


Whether R participated in an English as a 
second language program since 1986 


This question was dropped from the 
questionnaire. 




Collection of school information prior to 
1986 


The pretest instrument did not accept 
data if R attended a school prior to 1986 
that he/she had not informed us of in 
prior rounds. The instrument was 
revised to accept this information. 




Collection of job information 


The procedures for the collection of job 
information were altered somewhat. If R 
Indicated that he/she held the same job 
in two consecutive years, some of the 
questions asked in subsequent years were 
repetitive. Steps were taken to 
implement a skip pattern that skipped 
the repetitive questions. 



Data quality. The reinterview questionnaire repeated some of the questions that 
were asked in the Phase 5 pretest questionnaire, thus providing the basis for an 
assessment of data reHability. A number of measures of reHabiHty were computed for 
the 25 cases. Five of the items and their reliability measures are presented in Table 6.4. 
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TABLE 6 A — Results of interview/reinterview analyses 




1986 

% 
Match 


1986 
Correlation 


1700 

Absolute 
% 

Deviation 


1990 
% Match 


1990 
CorrelatioD 


1990 
Absolute 

% 

Deviation 


Number of Jobs 


92.3 






96.2 






Months Employed 


84.6 


.47 




96.2 


.88 




Hrs/Week Worked 


92.3 


.82 




92.3 


.97 




Months Unemvloyed 


88.5 


-.01 




96.2 


.69 




Annual Income 


53.8 


.77 


19.88 


38.5 


.99 


6.3 



It should be noted that the percent match statistic indicates the presence of an exact 
match. In general, data were more reliable for 1990 than for 1986. The relatively low 
correlation for reporting of months employed in 1986, along with its modest match quotient, 
suggests that respondents may have a difficult time recalling this information. Reliability for 
the same type of information for 1990 seems much better. For months unemployed in 1986, 
the low correlation is due to the item's extremely small variance and the presence of one or 
two large differences. For aimual income, exact match is low for both years, but the 
correlations are high, probably indicating differences in the way the respondents rounded 
income amounts during the two interviews. Judging by the correlations, reliability for 
income reporting is higher in 1990 than in 1986. 

Additional statistics were computed for income. An absolute percent deviation was 
calculated by dividing the absolute value of the difference between Phase 5 and reinterview 
income reports by the Phase 5 report. Results are shown in Table 6.4. On average, reports 
deviated by nearly 20 percent, in 1986 and by a little more than 6 percent in 1990. An 
average absolute deviation was also calculated. The average absolute deviation was $2,639 
(standard deviation, $4594) for 1986, and $1,132 (standard deviation, $1449) for 1990. The 
deviation for 1990 may be viewed as acceptable when one considers that 68 percent of 
reinterview respondents indicated that their income reports were accurate to the neai'est 
$1,000. However, the larger deviation for 1986 may indicate that the 1986 income reports 
are of poorer quality than those for 1990. 

The straight difference in income reporting (pretest minus reinterview) is -$997 
(standard deviation, $5,231) for 1986 and -$196 (standard deviation, $1,842) for 1990, 
suggesting either a positive bias during the pretest or a negative bias during the reinterview. 
Given the fact that the reinterview occurred without the benefit of substantial employment 
and income context set by numerous prior questions in the actual interview, it seems likely 
that problems in 1986 income reporting occurred during the reinterview. 
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During the reinterview, respondents were again asked whether or not they had 
taken courses for credit since March 1986 and the number of schools beyond high school 
they had attended since March 1986. For course taking all but three (88.5%) reinterview 
respondents gave the same response as during the Phase 5 pretest interview. Similarly, 
92.3 percent of respondents indicated the same number of schools in the reinterview. 

7. Field Test Training 

Four field test interviewers received 8 hours of project specific training on 
September 5 and 6. The main focus of the training was to aid interviewers in developing 
the skills needed to conduct an interview in the conversational manner required by the 
High School and Beyond fourth follow-up interview, and to teach the new skills needed 
to carry out other specialized functions, such as conducting on-line Federal Interagency 
Committee on Education (FICE) coding and using on-line question-by-question help 
specifications. 

The training was developed to include modules consisting of lecture, 
demonstration, and hands-on experience using the CATI system. Modules were 
developed to cover the following subjects: project overview, gaining cooperation, 
reviewing the question-by-question Specifications, using summary screens/developing 
conversational interviewing skills, on-line FICE coding. As part of the training, 
interviewers participated in two mock interviews. 

8. Interviewer Monitoring 

The HS&B:92 field test featured two innovations in interviewer monitoring: (1) 
optional remote site monitoring, and (2) a computerized monitoring system designed as a 
tool for statistical quality control. The remote site monitoring feature enabled NCES to 
monitor interviewers from Washington, and enabled HS&B management staff to monitor 
cases from their offices. This was in addifion to the monitoring done by interviewer 
supervisors as part of the statistical quality control plan discussed in the following 
section. 

The computerized monitoring/statistical quality control plan for the HS&B:92 
field test had three components: (1) an on-line monitor coding module, (2) a program to 
randomly sample interviewer behavior, and (3) a coding system for capturing interviewer 
behavior. 

On-Hne monitoring module. An on-line monitoring module was developed to 
allow monitors to enter monitoring codes into the computer, rather than recording them 
on paper. Use of this program will enable the provision of timely data on interviewer 
performance. 
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The first screen in this application allows the monitor to choose the mode of data 
collection. The second screen assigns an AutoQuest identification number to each 
monitoring session. The third screen asks monitors to enter their ID number and 
displays the current time and date. These are used to reconcile the monitoring 
assignments with the actual monitoring data collected. The fourth screen asks for the ID 
of the interviewer being monitored and the status of the station (HS&B interviewing, 
other HS&B activity, in use for another project, or vacant). The monitoring screens 
appear only when the monitor selects the HS&B interviewing option. When the other 
options are chosen the program terminates. 

The fifth screen can be reached a number of times in the course of a single 
monitoring session. This screen captures the activity currently being monitored. 
Activities are one of four types: Interviewing (QUEX), Locating (LOC), or Gaining 
Access/Coope ration (CO-OP). Because a single monitoring period may see each of 
these, some more than once, the program is designed to bring the monitor back to this 
screen each time he or she finishes a monitoring activity. The monitor also returns to 
this screen at the end of the monitoring session to select EXIT to finish the session. 

If Interviewing is selected as the current activity, the monitor receives a screen 
capturing the question number and judgments about the activity being monitored. 
Finally, when the monitor indicates that the monitoring session is finished, a screen 
capturing summary ratings appears. 

Statistical sampling of interviewer behavior. Monitoring periods were arbitrarily 
set to be fifteen minutes in length. A computer program was developed to randomly 
select from among all CATI stations designated for HS&B use. The program selects 
stations and allocates them at random, to fifteen minute time segments within a shift. In 
addition, the number of minutes between monitoring sessions was programmed to vary 
between eight and 15. The number of 15 minute monitoring periods selected 
corresponded to about 10 percent of the total amount of time interviewers were 
scheduled to work on HS&B during a given shift. 

At the beginning of a shift, the monitor was given a list of the stations to monitor 
during each fifteen minute monitoring period selected for that shift. Figure 1 shows a 
sample monitoring assignment sheet. Tlie assignment sheet assumes that the shift begins 
at 8:00 AM, ends at 4:30 PM, and that there are five stations available for High School 
and Beyond CATI interviewing. The first column allows the monitor to check off 
sessions as they are completed. The second column shows which station to monitor for 
each monitoring session. The third column provides the telephone number of the 
station, so that both the screen and the phone line can be monitored. 
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FIGURE 8.1 Example of a monitor assignment slieet. 



Schedule for: MONOOl 
Project: 4532 

Date: Thursday August 15, 1991 



Station ID Phone Ext Start Time Stop Time 



0001 7021 08:12 08:27 

0005 7025 08:38 08:53 

0001 7021 09:10 09:25 

0004 7024 09:36 09:51 

0005 7025 10:07 10:22 

0004 7024 10:33 10:48 

0005 7025 11:02 11:17 



Lunch 11:27 12:14 



0005 7025 12:15 12:30 

0005 7025 12:45 13:00 

0000 7020 13:16 13:31 

0000 7020 13:44 13:59 
0005 7025 14:10 14:25 
0005 7025 14:36 14:51 

0001 7021 15:03 15:18 
0000 7020 15:33 15:48 
0005 7025 16:02 16:17 



The fourth and fifth columns show when to begin and when to end each 
monitoring session. According to the schedule presented in Figure 1, the monitor would 
begin monitoring station 0001 at 8:12 and end that monitoring session at 8:27. The 
monitor would have eleven minutes between the first and second monitoring periods 
before he or she had to monitor again. At 8:38, the monitor would begin monitoring 
station 0005, and continue to do so until 8:53. Monitoring would continue through the 
day with a pre-determined lunch period which randomly varied in length. 
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Coding interviewer behavior. Two basic types of interviewer activities were 
monito red-interviewing, and gaining respondent access and cooperation. Interviewing 
was further divided into two subactivities-interviewing from scripted questions (including 
during the locating module of the instrument), and interviewing from unscripted 
summary screens. The following dimensions formed the basis for evaluating the quality 
of interviewer performance on scripted questions: question reading, response category 
reading, question repetition, providing definitions, probing, and response recording. 

For unscripted summary screens, the judgment dimensions were verification of 
information on the screen, verification of missing information, encouraging respondent 
use of their HS&B calendar, probing/clarification, and response recording. For gaining 
cooperation interviewers were judged on each of four dimensions: presentation of 
information about the study and it's sponsors, informing respondent about confidentiality 
safeguards, presentation of information about the contractor, and quality of interaction 
with the respondent. At the end of each monitoring session, interviewers were rated on 
their reading pace (too slow/just right/too fast), reading cadence (choppy versus fluid), 
and voice quality (stiff/formal/boring/uninteresting versus 
natural/conversational/interesting). Finally, interviewers were given an overall 
performance rating. 

For each question, the monitor coded an N, indicating no problems on these 
dimensions, an E (for error), a D (for deviation), or B, indicating that both an error and 
a deviation occurred. The criteria for judging E was that the interviewer did not follow 
the script (or standard procedures) on at least one of the dimensions and as a result may 
have adversely affected the response. The criteria for judging D was that the interviewer 
did not follow the script (or standard procedures) on at least one of the dimensions but 
the response was not likely to be negatively affected (or may have been improved). 

9. Field Test 

Sample design and selection. The field sample consisted of 110 selections from 
the 14,659 cases who were eligible for the third follow-up. To test the new instalment 
on cases with varying employment and educational histories, we again stratified the third 
follow-up sample using data from the third follow-up questionnaire. As with the pretest 
sample, we classified cases into four groups: those with two or more jobs and at least one 
education spell in the third follow-up; those with two or more jobs but no education 
spells in the third follow-up; those with one or more education spells but no jobs; and 
those with no education spells and fewer than two jobs. 

Prior to sampling, we sorted all 14,659 third follow-up respondents by group and, 
then within each group, we sorted the individual cases in a random order. We then 
selected a systematic sample of 110 cases from the sorted list. One case was 
subsequently dropped because it had already been selected for a pretest sample, leaving 
a total of 109. 
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Nonresponse analysis. One hundred and nine cases were selected for the field 
test sample, of which 73 (or 66.1 percent) completed the interview within the limited 4 
weeks of the data collection period. After 8 weeks, 80 (or 73 percent) of the cases were 
completed. Even though a longer collection period is expected to substantially improve 
response rates, it is important to determine whether nonresponse bias is likely to distort 
the results. In particular, it is important to determine how estimates concerning the 
average time required for various sections of the questionnaire may be affected by 
nonresponse. This section examines the correlates of field test nonresponse based on the 
73 cases completed within four weeks. 

Table 9.1 below displays the response rates for various subgroups of the field test 
sample. Tlie subgroups were defined using data from the third follow-up or earlier 
rounds. Two variables had a significant relation to nonresponse-educational attainment 
(Chi-square with 4 df= 10.14, p<.05) and whether information on the case's address had 
been obtained in the most recent updating effort (Chi-squaro with 1 df= 7.62, p<.01). 
The relation between field test nonresponse and educational attainment largely reflects 
the particularly low response rate among the nine cases who had still not received high 
school diplomas at the time of the third follow-up (1986); only two of these nine cases 
completed the field test interview. Apart from the high school dropouts, there was no 
apparent relation between nonresponse and educational attainment. 



TABLE 9.1 - Field PurticipuUon Rates by Selected Characteristics 



Variable 


Groups 


Respoase Rate 


u 






(in percent) 




Sex 


Male 


58.7 


46 




Female 


66.1 


63 


Race 


Black 


6L9 


21 




White 


69.7 


66 




Other 


59.1 


22 


Child nay 


No children 


65.0 


100 




At least one child 


77.8 


9 


SES Quartile 


Lowest Qua 'tile 


72.0 


25 




2nd Quartile 


60,9 


23 




3rd Quartile 


78.9 


19 




Highest Quartile 


67.6 


37 


EdiicatiomU 


No HS Diploma 


22.2 


9 


Attaiumeut 


HS Graduate 


72.6 


73 




License, AA Degree 


71.4 


14 




Bachelor's 


70.0 


10 


Address lufonuatiou 


Updated 


82.5 


40 




Not Updated 


56.5 


69 


Status ill 


Participant 


69.3 


101 


Third FU 


Nonparticipant 


25.0 


8 


Marital Status 


Never Married 


70.1 


77 




Married 


73.7 


19 




Separated 


40.0 


5 



27 



38 



The field test completion rate was also markedly lower among cases whose 
addresses had not been updated during the most recent updating effort (56.5%) than 
among those whose addresses had been updated (82.5%). Finally, there was also a 
marginally significant relationship between nonparticipation in the third follow-up and in 
the field test (Chi-square = 7.23 with 3 df, p < .07); of the 8 cases who were 
nonparticipants in the third follow-up, only two completed the field test interview. 

These results suggest that the field test results are likely to be representative of 
those for the fourth follow-up as a whole; the field test nonrespondents are exactly the 
same sorts of sample members (for example, dropouts, those with outdated locating 
information, and nonrespondents in earlier rounds) who are likely to be nonparticipants 
in the fourth follow-up. 

The key issue is whether the field test respondents provide a representative cross- 
section of those likely to complete the final fourth follow-up interview-in particular, 
whether the field test respondents are similar to the sample as a whole in terms of the 
complexity of their educational and employment histories. The field test sample was 
selected to include persons who had reported different levels of employment and 
educational activity in the third follow-up. Among those who had reported two or more 
jobs and one or more education spells in the third follow-up, the completion rate was 
quite high-84.6 percent (of 39 such cases selected); at the opposite* end of the spectrum, 
among those who had reported no more than one job and no education spells, the field 
test completion rate was much lower"48.0 percent (of 25 cases). Overall, there was a 
significant relation between field test parUcipation and field test sampling stratum, which 
was based on the number of jobs and education spells reported in the third follow-up 
(Chi-square = 10.96 with 3 df, p <.02). Because the number of jobs and education will 
be a major determinant of the length of the fourth follow-up interview, this final result 
suggests that the field test sample may over-represent cases with relatively long 
interviews. 

Cleaning of data collected in prior rounds. The field test included an initial trial 
of a new feature of the fourth follow-up questionnaire-the presentafion to the 
respondent of a summary of education and employment data from earlier rounds. A 
calendar with summary data on education spells and year by year data on employment 
was sent to the respondent before the field test interview; during the interview itself, this 
information was displayed to the interviewer, who reviewed it with the respondent. The 
respondent was given an opportunity to correct errors, fill in missing items, and update 
information that had changed since the last interview. The summary had several 
purposes. First, data on education spells had been created by NCES by linking 
information across earlier rounds; the summary review provided an opportunity to 
correct any errors introduced by the linking algorithm. Second, key items from earlier 
rounds were somefimes missing or the values that were obtained were sometimes 
implausible (e.g., wages rates). The summary review allowed us to fill in missing values 
and to correct obvious errors. Finally, the summary was a useful recall aid for collecting 
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information on education and employment since the last round, reminding respondents 
of events they had already reported and allowing them to link continuing education 
spells with spells that had been reported in earlier rounds. 

Was the summary review a useful data collection procedure? Table 92 suggests 
that it was; respondents often took the opportunity to change or add information in 
response to the summary review. For example, 63.5 percent of the respondents corrected 
information concerning their experiences at the first educational institution they attended 
after high school (see the third row below). 





TABLE 92 




Proportion of Respondents Changing 




Information During Summary Review 




Item 


Percent Reporting 


n 




Corrections 




Marital Status 


58.5 


41 


Status as HS Graduate 


0.0 


65 


Educational Experiences 






First Institution 


63.5 


52 


Second Institution 


54.2 


24 


Third Institution 


40.0 


10 


Employment Data 






Job Information for '83 


26,5 


66 


Job Information for *84 


29.4 


66 


Job Information for '85 


36.8 


66 



Educational data. To get a better sense of how the summary was used in the 
field test, we examined the type of changes made in the education data for the first 
institution attended after high school. The summary listed seven items for each 
institution-the location of the institution, the beginning and end date of the student's 
enrollment in the institution, enrollment status, field of study, degree sought, and 
whether that degree was obtained. A total of 35 respondents made changes in at least 
one of these items; altogether, these respondents made a total of 68 changes. Table 9.3 
shows how these changes were distributed across the individual items. 



29 



40 



TABLE 93 


Number of Changes Made In First Education 
Institution Data During Summary Review 


Item 


Number of Changes 


State/City 


2 


Stan Date 


3 


Ending Date 


17 


Enrollment Status 


1 


Field of Study 


11 


Degree Sought 


13 


Degree Attained 


21 



These changes fall into three major types. In some cases, respondents appeared 
to be correcting an error in the earlier data. The two changes in the location of the 
institution and the three changes in start date fall into this category. In other cases, 
respondents were providing information about an item that had previously been missing. 
Six changes were of this type (three involving the field of study, one the degree sought, 
and two whether a degree was attained). However, the vast bulk of the changes-52 of 
them-appear to represent new information that brings the earlier information up to 
date. For instance, 17 respondents reported changes in "end" dates of their enrollment; 
these all appear to have respondents who were in school at the time of the third follow- 
up and who now report they have completed spells at the institution. Similarly, ther 
were 8 respondents who reported changes in the degree sought, 8 who reported changes 
in the field of study, and 19 who reported attaining a degree that hcd not yet been 
completed at the time of the third follow-up; all of these changes are likely to represent 
attempts to update the third follow-up data. (The five remaining changes were not 
readily classifiable into one of the three categories.) These results suggest that the 
summary review was used more often for collecting new information than for correcting 
or filling in gaps in the existing data. 

Employment data. A similar analysis was conducted on employment data. Table 
9 4 lists the employment information that was verified and the changes that were made 
by year. 
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TABLE 9.4 

Number and Type of Changes Made in EmploymeDt Data 
During Summary Review by Year 


Year 


# Jobs 


Months 
Employed 


Hours per 
Week 


Months 
Unemployed 


Annuzd All 
Income Data 


1983 


0 


3 


5 


6 


7 


1 


1984 


0 


. 3 


3 


5 


9 


3 


1985 


1 


7 


5 


6 


11 


4 


Total 


1 


13 


13 


17 


27' 


8 



For months employed/unemployed, in 5 instances changes were due to the 
enforcement of a rule in the CAll program that did not allow the sum of the number of 
months employed and the number of months unemployed to exceed twelve. For these 5 
cases, the rule was violated in the preloaded data and corrected in the field test. It is 
also interesting to note that fifteen of the f;eventeen respondents who changed the 
number of months they were unemployed revised iheir reports downward. The pattern is 
not as clearly displayed among the changes to the number of months employed; nine of 
thirteen respondents revised their estimates downward. 

For hours per week employed, one pattern was identified in the revisions made to 
this item. In about half of the cases (8 of 13) the data entered during the field test 
replaced data that had been either missing or refused in the data from previous rounds. 
Several other corrections were made to items that had been incorrectly entered in 
previous rounds (e.g., 51 hours was changed to 15). 

For annual income, as reported in the field test CATI evaluation section, it 
appears that the wrong data may have been preloaded for this variable. We suspect that 
total household income was preloaded, rather than the respondent's individual income. 
The changes made to the preloaded data seem to support that hypothesis. About one 
half of the respondents (14 of 27) who changed the preloaded data for this item 
significantly reduced their reported income. Another four respondents provided income 
data for items that were either refused or missing in the preloaded data. The remaining 
respondents adjusted their annual income figures upward. 

There were eight instances when respondents changed all of the data items for a 
particular year. Whenever this occurred, the preloaded data contained only Os (as 
reported by the respondent during the 3rd follow up interview) or codes for item 
refusals, and the respondent provided updated data during the field test. Updates of this 
type were not included in the counts of updates to individual items. 
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Item frequencies. Item frequencies were examined with two major objectives in 
mind: (1) to assess item and response option usefulness, and (2) to spot problems with 
the CATl program. The ability to make item-by-item assessments of item usefulness was 
limited by the small number of respondents. Still, the following patterns were observed. 
First, the series of items on extracurricular activities in post-secondary institutions yielded 
only a few reports of participation in these activities (with the exception of employment). 
Second, the items that ask about the percentage of requirements fulfilled toward a 
graduate degree tends to elicit answers that are round numbers (e.g., 50 percent). This 
is a common problem with such items. Third, the items on ethnic composition also tend 
to elicit round numbers as answers. Fourth, as is commonly the case v/ith such items, the 
items on job satisfaction are highly skewed toward the satisfied end* Fifth, the items on 
various forms of political involvement tend to be skewed to the "never" end of the scale. 
These results will be considered when deciding which items, if any, will be cut from the 
main survey. 

Item frequencies were also examined to assess whether problems existed with the 
CAT! program A few minor puzzles were found. For example, the total number of 
people per household was captured as either 2 or missing. CATI program specifications 
for this item and for one or two other items with suspicious data will be carefully 
checked before the main survey. 

Section timings. The CATI program was designed such that the number of 
seconds that elapsed since the beginning of the interview was recorded in a number of 
places. Median timings for 18 sections of the questionnaire were calculated. The 
median timings were adjusted to account for the fact that the field test interviewers were 
judged to be considerably faster than average. Adjusted timings are presented in the 
following table. More will be said about the implications of the timings for the main 
study instrument in the recommendations section. 



TABLE 9.5 ~ Tuuings for sectious of the CATI iuterview 



Scc-^ 


Adjusted 




lion 


Minutes 


Topicf s) 


1 


0.853 


What were you doing last week? 


2 


0.769 


Household roster, marital status 


3 


2.897 


High school diploma, review of past ed. spells. 


4 


2.667 


New education spells 


5 


1.173 


Extracurricular activities 


6 


0.583 


Graduate and FS£ studies 


7 


1.033 


Educational aspirations, education loans 


8 


2.735 


Review of past job infonnation 


9 


15.410 


Collection of new job infonnation 


10 


0.203 


Armed Forces experiences 


11 


3.751 


Job training 


12 


2.324 


Job satisfaction 


13 


2.337 


Continuing education 


W 


1.442 


Children 


15 


1.679 


Values 


16 


3.603 


Group membership, political participation 


17 


4.009 


Income 


18 


0.666 


Race, racial composition 


Total 


48.133 





Monitoring Results 



For a number of reasons, the field test was not a good vehicle to use to conduct 
an extensive test of statistical monitoring. First, the sample was very small. Second, 
most of the cases were contacted and interviewed during the first weekend. During the 
remaining part of the field period in order to maximize coverage and accommodate 
appointments HS&B shop hours were scattered and irregular. Finally, cases worked 
after the initial weekend were difficult to locate and contact, making the likelihood that 
an interview would be conducted during the randomly selected monitoring period very 
small. Monitoring data were collected on a number of cases, but the number was quite 
small and, for the reasons listed above, the results are not indicative of overall quality of 
interviewing. 

Nonetheless, the field test did provide a good test of the monitoring systems. The 
hardware and software assembled for monitoring performed very well. Shop supervisors 
found it very easy to generate monitoring schedules. There was no difficulty in 
connecting to either the video or the voice portions of the interview. The monitoring 
data collection instrument was found to be easy to use and adequate to the task. The 
one failure was in attempting to present the monitor with both the video portion of the 
interview and the monitoring instrument on a single computer. This failure was due to 
incompafibility between the different software components used in monitoring and 
interviewing. Monitors instead used two computers side-by-side to watch and 
simultaneously record data. Alternate software is being explored for the main survey so 
that monitoring can be conducted using only one computer. 

One small improvement has been made to the monitoring instrument as the result 
of its trial in the field test. Rather than requiring the monitor to score each question as 
having either an error, a deviation, both, or neither, the program now only captures 
whether errors and/or deviations occurred. The "neither" category has been eliminated 
because it can be inferred by the absence of an error or deviation response. 

Effectiveness of the Calendar 

Results from the pretest reinterview indicated that about three-quarters of the 
reinterviewed respondents received the calendar, but less than half of those who received 
them thought that they were helpful during the interview. Questions about receipt of the 
calendar, its availability during the interview, and its usefulness were also included in the 
field test. Thirty-four of the seventy-three field test respondents (47%) said they 
received the calendar. Of those thirty-four who said they received the calendar, fifteen 
(44%) said they had it available during the interview and eighteen (52.9%) said they 
found the calendar to be useful. This low rate of availability h in spite of the fact that in 
the field test respondent letter we emphasized that respondents would need the 
calendars during interview. However, the judged usefulness of the calendar did not seem 
to depend on whether the respondent had it available during the interview. Of the 
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fifteen respondents who said they had received the calendar and had it available during 
the interview, 8 said it was useful and 7 said it was not useful. Of the nineteen 
respondents who said they had received the calendar but did not have it available to 
them during the interview, 10 said it was useful and 9 said it was not useful. 

Crude measures of the number of changes in past education and employment 
spells were created by summing across the items indicating whether each summary screen 
was correct. These measures were compared for those who had and who had not 
received the calendars. For past education data virtually no difference in the average 
number of screen changes was found (received calendar, mean =.7647; did not receive 
calendar, mean = .7692). For past employment data slightly more changes were made by 
respondents who did not receive the calendar (received calendar, mean = .7353; did not 
receive calendar, mean = 1.0513). This latter difference was not statistically significant 
(F(l,71) = 1.60). 

Locating 

Evaluation of NCOA* When individuals, households, or businesses change 
locations, they usually complete a Change of Address form notifying the United States 
Postal Service (USPS) of the move. The USPS compiles these change of address forms 
into a master file, called the National Change of Address (NCOA) database. Address 
changes remain in the NCOA database for three years and the file is updated every two 
weeks. In August, NORC subcontracted with a firm that is licensed by USPS to access 
the NCOA database. 

NORC provided the firm with the most recent names and addresses for every 
respondent, his/her parents, and up to 2 other contacts. In all, NORC submitted 53,128 
records to be checked against the NCOA database. 

The following summary statistics are based on all records submitted. Of the 
53,128 records processed: 

8.9% (or 4,758) records matched records in the NCOA database; 
84.3% (44,811) did not match records; 
6.6% (3,559) were identified as near matches.^ 

Before producing the field test case materials, NORC updated its records with 
information obtained from the 4,758 matches. 

While the firm could not provide change of address information on near matches, 
it did provide reasons why information could not be returned. For instance, in 2,640 



For a match to occur, each name and address must meet specific criteria. As a licensee, the firm may 
not provide change of address information if those, criteria are not met. 
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instances, a change of address could not be returned because the first name of the 
individual submitted by NORC did not match the first name of the individual in the 
NCOA database. In such a situation, a family member may have moved and is in the 
NCOA database, but the individual we seek is not. NORC plans to exhaust such leads 
with follow-up contacts to the local post office and former neighbors. 

Locating experiment. The design of the field test incorporated an experiment 
designed to determine the cost effectiveness of using an outside vendor for some of the 
locating work on High School and Beyond. Each case received first stage locating 
(enumerated below) by the interviewers before being classified as an "unlocatable" case 
and sent to a locating specialist. 

FIRST STAGE LOCATING ; 

• Calls to the respondent number listed on locator sheet 

• Calls to the parent number listed on locator sheet 

• Calls to other numbers listed for relatives/friends on the locator sheet 

• Calls to Directory Assistance to obtain information about the respondent 

At the end of the third week of field test data collection 41 cases remained 
unlocatable. These cases were divided at random into two groups; 21 were assigned to 
NORC locating specialists, and 20 were assigned to an outside vendor. At the 
beginning of the fourth week of data collection NORC locating specialists began to work 
on the "unlocatable" respondents in the NORC group. Respondents located by the 
locating specialists were immediately routed to an interviewer who attempted to contact 
and interview the respondent, thus the cases were worked on a flow basis. 

The unlocatable cases assigned to the locating staff in NORC's Telephone Center 
received second-stage locating. The custom tailored locating strategy used for the High 
School and Beyond fourth follow-up consisted of the following steps: 

SECOND STAGE LOCATING ; 

• Calls to Directory Assistance to obtain information about the respondent's 
parents 

• Calls to Directory Assistance to obtain information about the 
friends/relatives listed on the locator sheet 

• Credit bureau address update services 

• Library /CrissCross directories. 

• Respondent's high school 
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• College alumni and registrars offices (if respondent attended a 
postsecondary school) 

• Post office address update 

• Department of Motor Vehicles 

• Utility companies 

• Board of registrar 

• Social Security Administration 

• Welfare office 

The second group of 20 unlocatable cases was sent to an outside vendor for 
locating. In order to make it difficult to determine which persons on the list were High 
School and Beyond respondents the list was "salted" with cases which did not belong to 
the High School and Beyond sample. 

Results 

The NORC Telephone Center locating staff spent a total of 22.25 hours on 
activities associated with locating the group of twenty-two unlocatable cases that they 
were assigned. Their total costs for all locating tasks was $163.43 (includes fringe but 
not overhead). During that time they were able to locate eleven cases (50%) and 
interview 7 (31.8%) of those cases. Thus, locating cases in the Telephone Center 
required the following level of effort: 

Hours per located case: 2.0 
Percent of cases located: 50% 
Cost per case: $14.80 

The locating effort conducted by the outside vendor fell very short of initial 
expectations. Although the vendor had promised approximately a seventy percent "hit 
rate", only two (10%) of the twenty cases they were sent were fully located (address and 
telephone number provided and confirmed). Tliey were able to supply partial and 
unconfirmed data for other cases, but these cases then required additional locating work 
by the NORC Telephone Center. 

The resull is clear without even considering cost factors. The outside vendor was 
unable to deliver the level of service that they promised, and for this reason will not be 
considered for use on the main survey. Because the NORC Telephone Center was able 
to locate a larger proportion of cases than the outside vendor all locating work for the 
main High School and Beyond study will be conducted by the NORC Telephone Center. 
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10. Evaluation of Field Test CATI System 



The CATI system establis' id for use in the field test differed in several ways 
from other systems developed by NORC. First, in addition to collecting new data, 
interviewers were required to verify data collected in previous rounds of the study. As a 
result, the previously collected data was preloaded into the questionnaire so that it could 
be used by the interviewers. This process lead to the first of the innovations in the 
HS&B CATI system-reliance on a more conversational mode of interviewing. For the 
questions that verified past information, interviewers were not provided with a scripted 
question^ but rather were asked to elicit responses in a more conversational manner. 
Also, the HS&B CATI program required the interviewers to assign appropriate FICE 
codes to each new school for which they collected data. Finally, the CAIl program 
provided interviewers with online question-by-question specifications (q-by-q's) for 
questions that were difficult to administer. 

At the conclusion of the field test data collection period, a formal debriefing 
session was held to solicit feedback about the CATI system and the interview process 
from the interviewers. During that debriefing, intervievyers were asked to provide 
feedback about the interview in general, the new procedures (e.g., conversational 
interviewing, online FICE coding, etc.), specific items in the questionnaire, and the 
training procedures. The results of this debriefing are surxmiarized below. 

General CATI structure. The structure of the field test CATI instrument differed 
in some ways from the main instrument CATI system. As a matter of convenience, field 
test interviewers were asked to perform some tasks that will be automated for the main 
survey. For instance, the field test CATI instiiament relied on the interviewer to move 
from a window containing the locating and respondent verification portions of the 
questionnaire to the window containing the actual body of the questionnaire. 

/\11 of the problems that interviewers reported with the general structure of the 
CATI instrument are problems that are easily resolved by the implementation of the 
main survey CATI instrum^ent. For instance, the locator and respondent verification 
portions of the instrument will be contained in the Telephone Number Management 
System (TNMS) portion of the main survey CATI instrument. This greatly simplifies the 
process for moving between this portion of the questionnaire and the actual body of the 
questionnaire. 

Conversation mode interviewing. Interviewers reported that they easily adapted 
to the conversation mode interviewing technique and that it aided the flow of the 
interview. In practice, the interviewers also adopted a more conversational form of 
interviewing in the locating and respondent verification portion of the questionnaire. 
Monitors of interviews reported improved rapport, better flow (than observed in other 
CATI systems), and low error rates. 
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Verification of previously collected employment data. Interviewers reported that 
much of the data in the preloaded employment questions were revised by the 
respondents in the field test. However, most of the changes were required in one 
particular data item-annual income. It appears as if the wrong variable was selected 
(i.e. the annual income for the respondent's entire household, rather than the 
respondent's individual annual income). However, interviewers found that procedures 
for changing data on these screens were very easy to follow and reported no problems in 
making the changes. 

Verification of previously collected education data. Interviewers also reported 
that respondents did change a considerable amount of data on these screens. Again, the 
changes seemed to cluster into two variables. Respondents seldom changed data 
concerning a school name or location. The vast majority of the changes made were 
either in the dates of attendance or the major field of study. Once again, interviewers 
reported that the procedures for making such changes were simple and easy to follow. 

FICE Coding. In general, interviewers reported no problems with their ability to 
properly assign FICE codes to the schools for which we received information. However, 
interviewers did report several problems with the procedures in place for performing on- 
line FICE coding of schools. While the software functioned properly the majority of the 
time there were times when mechanical problems (software interactions) were 
encountered. Steps are being taken to minimize these problems for the main survey. 

Collection of new employment data. Generally speaking, this was the most 
problematic portion of the CAT! instrument. Interviewers reported that the flow of the 
interview was greatly slowed when respondents were asked to provide information about 
jobs since 1986. Because respondents were being asked to recall employment data for 
the past five years, this trend is rather easy to explain. However, the structure of the 
CATI program also played a role in bogging down the interview at this point. 

Because employment data is collected on an annual basis in the questionnaire, 
this section seemed very repetitive to respondents who have held the same job since 
1986. This problem was identified during the pretest interviewing and makeshift 
improvements were made for the field test. These makeshift changes were only 
somewhat effective. 

Preserving the data elements (employment status, earnings, industry /occupation, 
and training), the CATI screens for collection will be substantially streamlined (see 
chapter 11). 

School data. Interviewers reported no problems collecting data either on schools 
that were attended by respondents since the date of last interviewer or were attended 
earlier, but missed at the time of the last interview. In either case, this information was 
simple for the interviewers to collect. 
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Other data. Although interviewers did report minor problems with other items in 
the questionnaire, they reported no general problems in the collection of data for other 
questions. 

On-line question-by-question specifications (q-by-qs). As mentioned earlier, the 
CATI program also included on-line question-by-questions help screens for the 
interviewers to use as reference. Not every screen in the CATI program had a q-by-q 
attached. Rather, the q-by-q screens vere available for the following types of question 
screens: 

• Screens containing terms which needed clarification or definition. 

• Screens which asked interviewers to perform more difficult or unusual 
procedures (i.e. "code all that apply" questions) 

• Screens which contained questions that had set data collection rules to be 
enforced (i.e. in any given year the sum of number or months employed 
and number of months unemployed must be equal to or less than 12) 

Interviewers found the on-line q-by-qs very easy to use. The only complaint was 
that the q-by-qs were not comprehensive enough. During the interviewer debriefing 
session information was collected which would be used to supplement the q-by-qs for the 
main study. 

IL Modifications for Main Study 

This section summarizes findings from the field test and presents modifications 
adopted for the main study. 

Conversation mode interviewing. Both pretest and field test interviewers were 
unanimously comfortable with conversational mode. In addition to the flexibility needed 
to discuss data elements, the rapport with respondents seemed to be very positive. 
Finally, conversational mode interviews flowed well and were judged to be faster than 
rigid methods. These positive evaluations of the technique given by the interviewers 
were sufficient to extend its use beyond the main CATI to the introductory portion of 
the interview. 

Identification of respondents will be shifted to conversational mode, improving the 
timeliness of the process. 

Calendar. In addition to the extensive programming and producfion resources 
required to produce the calendars, they were infrequently used. Only about twenty 
percent of the field test sample had the calendar available during the telephone 
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interview. Further, findings concerning the accuracy of preload data on employment and 
earnings were disappointing. Therefore, the calendars were dropped from the main 
study. 

Locating. Almost 40 percent of the field test cases required additional (second 
stage) locating. Based on this result, it appears that a substantial locating effort will be 
necessary for the main study. Resources allocated for calendars will be diverted to 
locating. 

All locating work for the main survey will be completed by the staff in the NORC 
Telephone Center. The quality of the locating information received from the outside 
vendor during the field test locating experiment was inadequate and required further 
work by NORC staff. In contrast, the quality of the locating work performed by the 
NORC locators was very high. 

Before full-scale data collection begins, NORC will identify cases that have been 
classified as unlocatable in past rounds. These cases will go directly to locating 
specialists for second stage locating. In addition, the CATI interviewing staff will work 
through the main survey sample in such a way as to ensure that cases requiring second 
stage locating are identified as early in the field period as possible. 

Interview length and item refocusing. An important function of a field test was to 
assess the length of the interview. For HS&B:92 the target length for the interview was 
45 minutes. Since the field test CATI averaged 48 minutes, the following modifications 
will assure a proper level of burden during the main survey. 



Section: Verification and collection of postsecondury education spells 

Modifications Veri[y highest degree attained collected from all spell data, 

and rationale: 



Section: 
Modifications 
and rationale: 



Verification of information about jobs and earnings before 1986 
Rather than verifying all preloaded aggregate information, collect only missing 
items, or correct and/or verify out-of-range responses. The field test results 
indicated that most non-missing, within range preload information was correct. 



Section: 
Modifications 
and rationale: 



Collection of information about jobs since 1986. 

Collect employment/unemployment spells for the years 1986 to 1992. Several of 
the attributes of the job held longest each year were refocused. On-line industry 
and occupational coding will be used. Training topics were limited to those that 
occurred with sufficient frequency. 



Section: 
Modifications 
and rationale: 



Extracurricular activities 

Participation in varsity athletics, receipt of athletic scholarship, and the on-campus 
jobs during spells of postsecondary education were combined with the education 
spells. 
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Section: 
Modifications 
and rationale: 



Job satisfaction 

The section relating to opportunity for advancement, job security, and interpersonal 
relations were focused to make these items more pointed to current job. 



Section: 
Modifications 
and rationale: 



Values 

Refocus and reforma' the value items to agreement with major factors in values set. 



Section: 
Modifications 
and rationale: 



Political participation 

Reconfigiire the section to emphasize voter registration, voting behaviors in the 
1988 Presidential election and the 1990 Congressional elections. 



Section: 
Modlflcations 
and rationale: 



Income 

Reconfigure questions asking about sources of income for respondent and 
respondent's spouse. Refocus the questions on automobiles to emphasize the more 
directly useful question on balance of automobile loans as a measure of 
indebtedness. 



Preloads and recursive fills. With slight modifications the preloaded information 
proved to be relatively accurate for spells of education. The 1982-86 employment data 
were less accurate and verifications for missing and out-of-range values will be employed. 
Recursive fills within education and employment sections worked well and will be 
continued. 

Skips. Additional skips will be implemented for the full-scale CATL Based on 
the types of postsecondary institutions attended, data concerning extracurricular activities 
will be focused. Based on degree attainment, data concerning graduate/first-professional 
access and choice will be focused. Finally, based on labor force participation, data on 
employment will be focused. 

Data quality. Other than locating problems, which limit response rates, data 
quality seemed to be higher than for prior HS&B collections. In light of the complexity 
of the data elements, the choice of CATI rather than paper and pencil procedures, was 
well-justified. The number of skips, preloads, and recursive fills within the CATI would 
require substantially more burden if paper and pencil questionnaires were used. 

The location problems, due mainly to lack of contact with most respondents since 
1986, are formidable. Fortunately, telephone coverage appears to be quite good for the 
complete set of tracking information. The relatively short field period limited the 
contact rates and the longer period for the full-scale should improve response rates. If 
location/tracking resources limit overall response rates, subsampling procedures will 
ensure adequate effective response rates. 
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